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SPECIFICATION 

Electronic Version 1.2.8 
Stylesheet Version 1 .0 

Sensor Fusion System 
Architecture 

Background of Invention 

[0001] This Application claims the benefit of U.S. Provisional Application No. 60/366,439, 
filed on 03/21/2002. 

[0002] 1 . Field of the Invention 

[0003] The present invention relates to a collision avoidance system of a vehicle, and 
more particularly to a collision avoidance system of a vehicle that is capable of 
distinguishing the type of objects around a vehicle and the position and velocity of the 
objects relative to the vehicle. 

[0004] 2. Description of the Prior Art 

[0005] There is a continuing increase in the density of vehicles on the road and a 

simultaneous emphasis on improving the safety of highway and ordinary roadway 
vehicles operation by preventing vehicles from colliding with stationary and moving 
objects. One means for accomplishing this is to detect or monitor the relative speed, 
the direction of travel, and the distance between vehicles sharing the roadway, and to 
use such information to provide direct indications to the vehicle's driver of potential 
danger. 

[0006] For eX ample, US Patent 6,037,860 issued March 1 4, 2000 to Zander et al. and 
assigned to Volkswagen AG, teaches using a seat occupancy sensor to evaluate 
various collision strategies as an indicating function of possible severity of damage or 
injury. For this purpose, a collision strategy in which there are no injuries to persons 
is preferably selected, i.e. a collision with an inanimate object is carried out if this 
does not involve the driver and further vehicle occupants being subjected to excessive 
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danger. For this purpose, the motor vehicle is actuated in such a way that most of the 
unoccupied areas of the vehicle absorb the impact energy, i.e. the vehicle is turned 
with the unoccupied areas towards the object with which it is colliding. 

[0007] The Zander et al. system provides for a representation of the vehicle surroundings 
as determined by a number of laser sensors, radar sensors and thermal imaging and 
other cameras or a combination thereof. Thus, the Zander et al. system detects 
moving objects in the environment and internal vehicle conditions, in order to 
generate vehicle controls to avoid or minimize vehicle collisions. Sensor information is 
processed in an evaluation unit to determine a course of action and vehicle actuators 
implement the selected course of action. It would be desirable to have a more 
complete list of information regarding type of objects, distance from the host 
(including radial and angular information), and relative velocity. In particular, it would 
be desirable to have object-tracking data for visual awareness purposes, threat 
analysis, and possible subsequent vehicle actions. These are some of the problems 
the present invention overcomes. 

[0008] Further, the Zander et al. patent generally alludes to combining of various sensor 
technologies to form a picture of the targets surrounding the vehicle. However, for 
any particular target at any particular time, there is no teaching of using more than 
one sensor to obtain target-tracking information. It would be desirable to teach 
combining sensor information in an explicit way such that each field around the host 
car is covered in duplicate by radar and a camera. Using a fusion combination, in 
accordance with an embodiment of this invention, the best and most useful data from 
the radar is combined with the best and most useful data from the camera. 

[0009] US Patent No. 5,479,173 issued December 26, 1995 to Yoshioka et al. and 

assigned to Mazda Motor Corporation teaches combining radar and camera data to 
determine a danger level of an object in the predicted path of the host car. It would 
still be desirable to build a complete picture of objects around the host car. 

[0010] U5 p atent ncl 5,475,494 issued December 12, 1995 to Nishida et al. and assigned 
to Mitsubishi Denki Kabushiki Kaisha teaches gauging size and distance of objects 
based on a calculated two-dimensional image. The system looks forward only and 
does not use true radar sensing. Data from a single forward looking camera are 
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combined with data from a set of laser radar sensors. 

Summary of Invention 

[001 1] This invention uses complementary sensors to provide information characterizing 
the environment around a vehicle. This information can be further processed and used 
in a variety of ways. For example, there may be implemented new advanced electronic 
features that use remote sensing in an integrated and cost-effective manner. Such 
features may include lane changing aids, night vision assistance, lane departure 
conditions, and other such features. 

[0012] Present solutions to providing information characterizing the environment around 
a vehicle often rely on single sensing technologies such as visual cameras, for lane 
departure, infrared (IR) cameras for night vision, and radar for adaptive cruise control. 
Such single sensing technologies are limited in robustness due to environmental and 
technological limitations. Such systems typically fail to take into account information 
gained by other sensors. 

[001 3] In accordance with an embodiment of this invention, an electrical architecture 

integrates, or fuses, multiple sensors, and multiple sensing technologies, to create a 
data bank of information on objects surrounding the vehicle. Fused data can identify 
positions and velocities of objects, and can classify them in rough categories 
according to object size, such as heavy truck, passenger car, motorcycle, pedestrian, 
and so on. These data can then be communicated to all relevant control modules in 
the vehicle, to allow for more intelligent implementation of advanced features. It is 
particularly advantageous to combine sensors from two different bands of the 
electromagnetic spectrum. For example, visual cameras and radar sensors can be 
combined to achieve the above objective. Cameras and radars are located around the 
vehicle and sense overlapping areas. This allows distance data from radar to be 
combined with visual data from cameras. 

[0014] 

Thus, this invention teaches detecting the environment and providing a reliable 
set of data about objects in the environment. Radar and camera sensors can provide 
the inputs to a central fusion module. The central fusion module processes the inputs 
to generate object-tracking data. This includes a list of information regarding type of 
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object, distance from host car, including radial and angular information, and velocity. 
The object tracking data are available as output from the central fusion modules for 
visual awareness purposes, threat analysis, and possible vehicle actions. 

[001 5] In accordance with this invention, the sensors are arrayed such that each field 

around the host car is covered in duplicate by radar and a camera. The combining of 
the data is then done by fusion. That is, the best and most useful data from the radar 
is combined with the best and most useful data from the camera. Using both sensing 
technologies provides an improvement over using a single sensing technology alone 
to determine necessary information about surrounding objects, whether moving or 
stationary. This type of data fusion requires a system that provides a time tag with the 
data so that the data can be combined effectively. If desired, the data can be 
interpolated and extrapolated to the point in time for the fusion to provide increased 
accuracy. These fused data are stored in a target track file. Subsequent additional 
fused data are compared to the current track files and if a match is found the track file 
is updated. The current track files are output for further processing. 

Brief Description of Drawings 

[001 6] Figure 1 is a plan view of a vehicle with information gathering zones A, B, C, and 
D, each zone including two sensors. 

[0017] Figure 2 is a block diagram showing a collision avoidance system of a vehicle 
according to one embodiment of the present invention. 

[001 8] Figure 3 is a block diagram showing a more detailed view of the input sensors 
shown in Figure 2. 

[0019] Figure 4 is a block diagram of the function of the fusion control module as 

information from the sensors is processed to obtain target definition for multiple 
targets using matching of time tagged data. 

[0020] Figure 5 is a block diagram of the function of the fusion control module as 

information from the sensors is processed to obtain target definition for multiple 
targets using extrapolation and interpolation of time tagged data. 

Detailed Description 
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[0021] Figure 1 is a plan view of a vehicle 1 with illustrated sensing zones A, B, C, and D. 
In each of zones A, B, C, and D two sensors are deployed. The sensors are each from 
different bands of the electromagnetic radiation spectrum and provide complementary 
information. Advantageously, one is selected to be an active radar sensor and the 
other a visual sensor, such as a camera. Thus, there are two overlapping sensors in 
each of the zones A, B, C, and D. Because the sensors overlap complementary 
information is available about any target in the zone. The two sensors can provide 
more information about the target than either sensor could individually. More 
specifically, zone A includes a camera sensor 110 and a radar sensor 210, zone B 
includes a camera sensor 1 1 1 and a radar sensor 21 1 , zone C includes a camera 
sensor 1 1 2 and a radar sensor 212, and zone D includes a camera sensor 1 1 3 and a 
radar sensor 21 3. 

[0022] Information from these camera sensors 1 1 0, 1 1 1 , 1 1 2 1 1 3 and radar sensors 2 1 0, 
21 1 , 21 2, 21 3 is processed as complementary pairs to determine information about 
the environment surrounding the vehicle. Information from the sensors, camera 
sensors 1 1 0, 1 1 1 , 1 1 2 1 1 3 and radar sensors 210, 211, 212, 213, is time tagged to 
indicate the time that the information was gathered. In accordance with an 
embodiment of this invention, information from paired camera sensor 1 10 and radar 
sensor 21 0 in zone A with the closest time tags is compared and fused to provide 
fused information data. This information is then used to generate characterizing 
descriptions of any target in the particular zone generating the sensor signals. 
Similarly, information from sensor pairs in zone B (camera sensor 1 1 1 and radar 
sensor 211), zone C (camera sensor 1 12 and radar sensor 212), and zone D (camera 
sensor 1 1 3 and radar sensor 21 3) is time tagged and fused to provide fused 
information data. For example, radar sensor 2 1 0 and camera sensor 1 1 0 in zone A 
can provide information about a motorcycle in zone A. Such information can include 
size, speed, direction, acceleration, and so on. 

[0023] Radar sensors are particularly advantageous to help determine distance and 

whether a target is moving or standing still. Relative velocity can be determined based 
on the frequency shift of the received radar signal versus the transmitted radar signal. 
Cameras can provide good information with respect to angular information and object 
recognition. Thus the complementary combination of a camera and radar sensor 
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provides more information than either sensor could individually. 

[0024] Figure 2 is a block diagram showing a vehicle collision avoidance system 1 0 of a 
vehicle, according to a preferred embodiment of the present invention. Collision 
avoidance system 10 includes an input detecting section 1 1 having an output coupled 
to a central fusion-processing module 13. The output of central fusion module 13 is 
applied to a vehicle control module 15. Central fusion module 1 3 supplies an 
activating output to vehicle cpntrol module 1 5, which may include a plurality of 
vehicle control modules. In situations where a crash may be imminent, an active safety 
system may be energized and other pre-crash action may be taken. Thus an active 
and a passive safety module may be activated. In turn, these modules may control a 
braking module, a light control module, an adaptive steering module, a security 
module, and other modules. 

[0025] More specifically, referring to Figure 3, input-detecting section 1 1 includes four 
visual cameras 110, 111, 112, and 1 1 3 and four radar sensors 210, 211, 212, and 
213. There is an output from the combination of sensors in input detecting section 1 1 
to central fusion module 1 3. The visual cameras give good angular information about 
adjacent objects. The radar sensors give good depth and distance information about 
adjacent objects. 

[0026] Referring to Figure 4, input-detecting section 1 1 provides inputs to central fusion 
module 13. Central fusion module 13 includes a tag block 21 coupled to receive an 
output from input-detecting section 1 1 for adding a time tag to the incoming sensor 
signal. Thus, each sensor signal is identified with respect to the time the sensor signal 
is received. Tag block 21 has an output coupled to the input of a matching block 22. 
Matching block 22 matches the time tagged visual and camera sensor inputs to a time 
tagged radar sensor input. Advantageously, this is done so that the closest time tags 
of each of the visual and radar inputs are matched. The output of matching block 22 
is applied to a target definitions block 23. 

[0027] Thus, referring to Figure 4, a time tag indicating the time the data is generated is 
added to the input data received from the input sensors. Advantageously, some 
information from all the input sensors is used. A qualitative determination is made 
with respect to the value and accuracy of the information from each sensor. The best- 



APP ID=10065958 



Page 6 of 18 



3 iS t U ini ll m !hi K :H, r*!! 11, 3 n 4 O irl! 



input data is weighted more heavily than input data of lesser reliability. Thus, for each 
target a time tagged compilation of input data is used to characterize the target with 
respect to position, velocity, acceleration, size, and so on. This same procedure of 
using data from all sensors is applied to all targets in the surrounding environment. 
Further, for each target tracking data, the process is repeated as time goes on so that 
an updated target track is continually generated. 

[0028] Referring to Figure 5, input-detecting section 1 1 provide inputs to central fusion 
module 1 3a. Central fusion module 1 3a receives an input at a tag block 21a. Tag 
block 2 1 a has an output coupled to an interpolation/extrapolation block 22a. 
Interpolation/extrapolation block 22a processes information to get time/target 
information of increased accuracy at those points in time when actual time tagged 
sensor information indicates a slightly different time. To compensate for these slight 
differences in time the information from a sensor pair is interpolated and/or 
extrapolated to generate information characterizing the environment around the 
vehicle at substantially the same time. That is, after the data is time tagged it can be 
used to interpolate target information between time tags, and can be extrapolated to 
indicate projected target information after a time tag. The output of 
interpolation/extrapolation block 22a is coupled to target definitions block 23a. 

[0029] In summary, a fusion process combining complementary data is used to 

characterize each target. The fusion process uses data from different sensors at the 
same point in time to get better definition of target objects surrounding the vehicle. 
The information characterizing each target is then combined to generate a picture of 
the environment surrounding the vehicle. Such a picture would include all the targets, 
in all directions from the vehicle. Thus, the system provides a time tag with the data 
so it can be extrapolated to the same point in time for the fusion process to take 
place. The fused data can be stored in a target track file. Additional subsequent fused 
data can be compared to the current track files and if a match to a target is found the 
track file is updated. The current track files can be output for further processing. 

[0030] 

Target-tracking information is obtained from all sensors of this embodiment. Each 
piece of information from a sensor relating to a particular target is given a time tag to 
identify it in the time domain. A plurality of such target tracking pieces of information 
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is used through interpolation and extrapolation to obtain additional target tracking 
pieces of information. Such information from all sensors is fused together to get the 
best representation of the target position at a given time. This process is repeated for 
all targets around the 360-degree circumference of the vehicle. The totality of these 
targets is then combined to form a total picture of targets surrounding the vehicle. 

[0031] While the present invention has been particularly shown and described with 

reference to preferred embodiment thereof, it will be understood by those skilled in 
the art that the foregoing and other changes in form and details may be made therein 
without departing from the spirit and scope of the present invention. For example, 
infrared sensors and laser sensors may be selected for use from the electromagnetic 
spectrum. Also, the time tagging function may occur at the sensor if the central fusion 
module has distributed processing ability. 
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